Phase Diagram of the Kane-Mele-Hubbard model 
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Motivated by recent numerical results, we study the phase diagram of the Kane-Mele-Hubbard 
(KHM) model, especially the nature of its quantum critical points. The phase diagram of the Kane- 
Mele-Hubbard model can be understood by breaking the SO(4) symmetry of our previous work 
down to U(l)spin X U(l)chargc X PH Symmetry. The vortices of the inplane Neel phase carry charge, 
and the proliferation of the charged magnetic vortex drives the transition between the inplane Neel 
phase and the QSH insulator phase; this transition belongs to the 3d XY universality class. The 
transition between the liquid phase and the inplane Neel phase is an anisotropic 0(4) transition, 
which eventually becomes first order due to quantum fluctuation. The liquid-QSH transition is 
predicted to be first order based on a 1/N calculation. 

PACS numbers: 



I. INTRODUCTION 

Thanks to the discovery of Grapheneir— , a great deal 
of attention has been devoted to systems with Dirac 
fermions at low energy. It was demonstrated that 
many topological states of fermions are related to Dirac 
fermions, such as the quantum Hall stated, quantum 
spin Hall state^i^, and 3d Topological insulator— i^, etc. 
Since last year, motivated by the quantum Monte Carlo 
(QMC) simulation on the Hubbard model on the honey- 
comb lattice^, strongly interacting Dirac fermions have 
stimulated a lot of interests. Quite unexpectedly, a fully 
gapped liquid phase was discovered in the phase diagram 
of the honeycomb lattice Hubbard model at intermedi- 
ate Hubbard (7—, and by increasing U this liquid phase 
is driven into a Neel phase after a continuous quantum 
phase transition. This liquid phase has stimulated many 
theoretical and numerical studies on possible spin liquid 
phases on the honeycomb latticei^ii. So far almost all 
the theoretical proposals about this liquid phase involve 
nontrivial topological ordersiSr— li^ii^. 

The Hubbard model on the honeycomb lattice has the 
full S0(4) ^ [SU(2)spin X SU(2)chargc]/^2 symmctryi^^, 
thus a true liquid phase of the Hubbard model should pre- 
serve all these symmetries. In Ref. — li^, a full SO (4) in- 
variant theory of the Hubbard model was developed, and 
it was proposed that the liquid phase observed in Ref<^ 
is a topological spin-charge liquid phase with mutual 
semion statistics between gapped spin-1/2 and charge— e 
excitations. The global phase diagram of this theory is 
depicted in Fig. [TJ We will review this theory in the next 
section. 

In the current work, we will consider perturbations 
on the Hubbard model that break the SO (4) symmetry 
down to its subgroups. In particular we will focus on 
the Kane-Mele-Hubbard model that was recently stud- 
ied numerically22ri2^: 
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The second term of this Hamiltonian is the spin-orbit 
coupling introduced in the original Kane-Mele model for 
the quantum spin Hall effect (QSH)--i^. The goal of the 
current work is to understand the change of the phase 
diagram and quantum critical points due to the existence 
of the QSH spin-orbit coupling, compared with the SO (4) 
invariant case. 



II. SO (4) INVARIANT THEORY 

In Refji^, the author used the S0(4) symmetry to clas- 
sify the order parameters on the honeycomb latticc^S. 
In particular, the quantum spin Hall (QSH) and triplet- 
superconductor (TSC) order parameters belong to a 
(3, 3) matrix representation of the SO (4) group: 

/ Im(TSC):r , Im(TSC)y , Im(TSC)^ \ 



Re(TSC)^ , Re(TSC)j, , Re(TSC), 



V QSH, 



QSH„ 
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Since all these order parameters are topolog- 
ical, their topological defects carry nontrivial 
quantum numbers. For instance, a Skyrmion 

of the spin vector (QSHx, QSHy, QSH^) carries 
charge— 2e2^, while a Skyrmion of the charge vec- 
tor (Im(TSC)z,Re(TSC)z,QSHz) carries spin-1 i.e. 
spin and charge are dual to each other, and view each 
other as topological defects-i^. The liquid phase proposed 
in Ref.^^'^^ was obtained by proliferating both the spin 
and charge Skyrmions from the condensate of Qab- 

In the S0(4) invariant theory, the low energy physics 
of topological defects of the matrix order parameter Qat 
is described by the following theoryJ^^: 
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FIG. 1: The global phase diagram for the SO(4) invariant 
theory Eq. [3] proposed in Ref.^^. 



+ \id,-zAl\z^J'+r,\z^J' + 
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The CP(1) fields z^ and z^ are SU(2)spi„ and SU(2)chargo 
fundamental doublets. In terms of z^ and z^, the order 
parameter Qab is represented as 



Qab ^ (Z^V''Z^)(Z^VZ^). 



(4) 



The mutual Chern-Simons field in Eq.|3] identifies z* (z^) 
as the vortex (meron) of charge (spin) sectors of the order 
parameter Qab- 

By tuning the parameters r^ and Tc, a global phase 
diagram Fig.[T]is obtained. There are in total four phases: 

(1). Phase A3 corresponds to the case with z" and 
z"^ both gapped, and the system is in a topological liq- 
uid phase with mutual anyon statistics between spin— 1/2 
and charge— e excitations. This statistics is guaranteed 
by the mutual CS fields in Eq. [3] In Ref)^ it was pro- 
posed that this is the liquid phase observed by QMC— . 

(2). In Phase A2, z^ is condensed while z° is gapped. 
The system is in a magnetic ordered phase with both 
Neel and transverse nematic order. The ground state 
manifold (GSM) of this phase is SO(3)/Z2. 

(3). In phase A, both z" and z'^ are condensed, the 
system is described by the condensate of order parameter 
Qab, with GSM (52 X S'^)IZi. One example state of this 
phase is the QSH^ state, which couples to the fermions 
in the same way as the spin-orbit coupling introduced in 
the Kane-Mele model^i^. 

(J^). Phase A4 is the charge-dual of the phase A2, with 
the same GSM SO(3)/Z2. 

All the phase transitions in phase diagram Fig. [1] are 
continuous. For example, the transition between A3 
and A2 belongs to the 3d 0(4) universality class, while 
the transition between A and A2 is a CP(1) transition, 
which is equivalent to the deconfined quantum critical 
poin t? "i ' M^ . The multicritical point in Fig. [1] was studied 
using a 1/iV expansion in Refj^, and when N is suffi- 
ciently large this multicritical point is a conformal field 
theory. 



The same field theory Eq. [3]was used to describe vari- 
ous phases observed experimentally on the triangular lat- 
tice frustrated magnetsS^, such as k— (ET)2Cu2(CN)3, 
EtMe3Sb[Pd(dmit)2]2, EtMe3P[Pd(dmit)2]2, etc. Also, 
a similar theory without SU(2)chargc was applied to the 
cuprates^iiS^. 



III. THE KANE-MELE-HUBBARD MODEL 
A. General formalism 

In the Kane-Mele-Hubbard (KMH) model Eq. [U the 
symmetry of the Hubbard model is broken down to 



U(l) 



spin 



U(l)chargc X PH, 
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where U(l)spin is the spin rotation around z axis, while 
U(l)ciiargc corresponds to the ordinary charge U(l) rota- 
tion. The extra particle-hole symmetry (PH) is 
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This PH symmetry is in fact a product of spin and charge 
TT— rotations around the y axis: PH = tt^'"^ •tt^''*. One can 
verify that tt^'"^ and tt^'* individually changes the KMH 
model, while their product keeps the model invariant. 
The definition of PH is not unique. For instance, one 
can also define PH as PH — ■k^''^ ■ tt^'*, then fermion 
operator transforms as q -^ cr^c|(— 1)*. 

The QSH spin-orbit coupling corresponds to (533 of the 
matrix order parameter Qah in Eq. [2] Thus a nonzero 
{Q33) in the Hamiltonian will modify the field theory 
Eq. [3] as follows: 
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Notice that terms like z'^V^z'^, z'^'ct^z" etc. are all for- 
bidden by the PH symmetry. The symmetry-breaking in- 
troduced by the QSH spin-orbit coupling will not change 
the nature of the liquid phase (A3 of Fig. [Ij. However, 
the other phases with spin and charge orders will be mod- 
ified. The modified global phase diagram is depicted in 
Fig. Hi. 



B. Neel phase and charged vortex 

Let us start with phase A2. With the background QSH 
order parameter, the phase A2 of Fig. [1] is reduced to a 
pure inplane Neel phase with GSM S^ in Fig. [2h- 

One very special property of this Neel order is that, the 
vortex of the Neel order carries unit electric charge due 
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FIG. 2: (a). The global phase diagram of Eq. [71 for models 
that break SO(4) to U(l)spin x U(l)chargc xPH symmetry, (fe). 
The phase diagram of the actual KMH model. The theory in 
Ref.— would predict an extra transition line inside the Neel 
phase (dashed line), which corresponds to the order-disorder 
transition of the CAF order parameter. This dashed line is 
absent in our theory. 



to the "dual" QSH effect, since the vortex of the inplane 
Neel order carries a magnetic tt— flux. This "dual" QSH 
effect and charged spin-flux was discussed in Ref i^^'^° . 
The key question we want to address here is, does the 
charge carried by the vortex affect the quantum phase 
transitions around the Neel phase? 

This problem can be understood by classifying the 
charged vortices of the Neel order based on the symmetry 
of the system. Every charged vortex carries two quantum 
numbers, charge and vorticity, denoted as (e, w). There 
are in total four flavors of charged- vortices: 



{e,v), (zj)* = (-e,-w). 



(8) 



z1 and ^2 are precisely the SU(2)chargc doublet introduced 
in Eq. [31 If there is a full S0(4) symmetry, all four fla- 
vors of vortices are degenerate. Within the current KMH 
model, the symmetry guarantees that z^ and (z^)* are 
degenerate, but zj and z| are not necessarily degenerate. 
This is due to the fact that under the PH transformation 
in Eq. [HI both e and v are reversed. (Notice that PH 
defined in Eq. l transforms S"" ^ - S"" , Sy -^ Sv .) 

The classification of vortices can also be understood 
from the general theory Eq. [71 For instance, in phase 
A2 (condensate of z^), due to the existence of the last 
term of Eq. [S] the condensate of z^ splits the degeneracy 
between zj and Zg. 

This inplane Neel order is always accompanied with 
a background QSH spin-orbit coupling. Since the QSH 
spin-orbit coupling breaks the reflection symmetry x — ^ 
—X of the honeycomb lattice, one might expect that the 
following spin order parameter automatically acquires a 
nonzero expectation value: 



i^' ~ ^ i^ijz ■ {Si X Sj). 



(9) 



However, H' is odd under PH. Thus unless the system 
further breaks the PH symmetry, H' should not have any 
nonzero expectation value. 



C. Neel-QSH transition 

Usually the order-disorder transition of inplane XY or- 
der is driven by proliferating the vortices of the order pa- 
rameter. In the previous section we have classified the 
vortices in the inplane Neel phase. In the Neel phase, 
since vortices zf and z| are not degenerate, only one com- 
ponent of the vortex doublet z^ condenses at the tran- 



sition. Let us take this vortex to be z 
theory of the transition is 
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This is a 2+ Id Higgs transition, which belongs to the 
3d XY universality class. The gauge field A^ is pre- 
cisely the dual of the Goldstone mode of the inplane Neel 
phase. The condensate of zi has no Goldstone mode due 
to the Higgs mechanism, thus the condensate has no su- 
perconductor order even though zj carries charge. The 
condensate of zj is precisely the QSH insulator. 

If we start with the QSH insulator phase, this QSH- 
Neel phase transition can be viewed as condensation of 
magnetic exciton hi ~ cl ^c^,^. Since (6^)^ = 0, the 
magnetic exciton is a hard-core boson. Under PH trans- 
formation, bi transforms as hi -^ ^b*. This symme- 
try rules out the linear time-derivative term in the La- 
grangian of 6, thus this transition is an ordinary 3d XY 
transition, which is consistent with the analysis in the 
previous paragraph. 

As a comparison to the KMH model, let us discuss a 
slightly different kind of symmetry breaking of the Hub- 
bard model. In this case, the SO (4) symmetry of the 
Hubbard model is broken down to U(l)spin x U(l)chargo x 
^y,c ^ Ti-i/.s^ i.e. both tt^"'^ and tt^'^ are symmetries of 
the system individually (in the KMH case only their 
product is the symmetry). According to the symme- 
try U(l)spin X U(l)chaigc X 11^^'' X TT^^", in the inplane 
Neel phase (phase A2) all four flavors of charged vor- 
tices (merons) with quantum numbers (±e, ±w) are de- 
generate. Thus the low energy field theory describing 
these charged vortices is the CP(1) model with easy-plane 
anisotropy: 
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We keep m < 0, thus the SU(2)chargc is broken down to 
the easy-plane direction, i.e. zj and z| both condense at 
the transition. When z% both condense, the system en- 
ters a superconductor state with one Goldstone mode. 
This superconductor is the triplet superconductor TSC^ 
in the matrix order parameter Eq. [21 If we define an 



0(4) vector = (NeeL,Neelj„Re[TSC^],Im[TSC^]), the 
nonlinear sigma model of (f) has a topological 0— term. A 
Neel-TSC transition on the honeycomb lattice was dis- 
cussed in Ref."^^. However, this Neel-TSC transition does 
not happen in the KMH model, since in the KMH model 
zl and Zj are nondegenerate. 



D. Liquid- Neel transition 

In the KMH model, the phase transition between the 
spin-charge liquid phase and the Neel phase does not in- 
volve any low energy charge degrees of freedom, thus this 
transition can be understood as the condensation of z^, 
while z^ are gapped. When z^ are gapped out, they 
can be safely integrated out from Eq. [71 then zf and z| 
become degenerate, z* is coupled to a ^2 x ^2 gauge 
field, as was discussed in RefJ^i^. Now the liquid-Neel 
transition is described by the following Lagrangian 

2 2 



Q = l 



a = l 



'II 1^21 -r • • ■ (12) 

The Neel order parameter is a bilinear of z^: 

iV^ ~ Re[(z'^)*icrV^z''] - Rc[z2 - z^], 



Ny - Re[(z' 
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The first line of Eq. [TH has a full 0(4) symmetry, 
while the second line breaks the 0(4) symmetry down 
to U(l) X U(l) X Z2. In this field theory m > 0, thus in 
the condensate of z^ there is only one Goldstone mode 
that corresponds to the inplane Neel order. According 



to the high order e expansion in Ref 



34 



is a relevant 



perturbation at the 3d 0(4) universality class, which is 
expected to drive the transition first order eventually. 



E. The s— wave superconductor 

The phase AA in Fig.[T]is reduced to the s— wave super- 
conductor in Fig. [2jz. The s— wave SC is the charge-dual 
of the inplane Neel order in phase A2. Just like the mag- 
netic vortex of the Neel order, the vortex of the SC also 
carries two quantum numbers: spin-±l/2 and vorticity: 
{s^v). The symmetry of the KMH model divides the 
vortices into two groups: (i, v) and (— ^, —v) are degen- 
erate, while (i, — w) and (— i,w) are degenerate. These 
two groups of vortices are precisely (zf , z^*) and (z|, z|*) 
introduced in Eq. [S] 

The quantum phase transition between the s— wave SC 
(phase AA) and the QSH insulator (phase A) is inter- 
preted as condensing either zf or z|, this transition is 
a 3d XY transition. The transition between the liquid 
phase ^3 and phase A4 is described by a similar theory 
as Eq. [T21 and it is expected to be a first order transition. 



F. Multicritical point 

There is a multicritical point in our phase diagrams 
Fig- ID for S0(4) invariant systems, which separates the 
liquid phase from the QSH insulator. In Ref,-?-?, this mul- 
ticritical point was studied using a large— iV generaliza- 
tion, where TV is the number of components of z^ and z^. 
It was demonstrated that when N is large enough, this 
multicritical point is a conformal field theory2£. Com- 
pared with the theory studied in Ref.^^, Eg. [7] has several 
SO (4) symmetry breaking perturbations, for example the 
last term in Eq. [T] If we extrapolate the 1/A^ calculation 
in Ref.-S to our current case with TV = 2, the last term 
in Eq. [7] is a relevant perturbation at the multicritical 
conformal field theory. Thus we expect the transition 
between the liquid and the QSH insulator in the KMH 
model to be a first order transition. 



IV. COMPARE WITH OTHER THEORIES 

Based on our theory discussed in this paper, we pro- 
pose the phase diagram for the KMH model in Fig. ^ 
plotted against A and U . When A = 0, there is one extra 
transition inside the Neel phase, which corresponds to the 
transition between the pure Neel order with GSM S"^ , and 
the Neel + nematic order with GSM SO(3)/Z2 (phase 
A2 in Fig. [T]). This transition is absent once nonzero A 
is turned on, this is because the symmetry of the Neel -|- 
nematic order order is identical to the Neel order with a 
background QSH spin-orbit coupling. 

In Ref.Ai, the authors proposed a different phase dia- 
gram for the Hubbard model on the honeycomb lattice. 
Instead of a Neel-|-nematic order, the authors of Ref.— 
predicted a chiral antiferromagnetic (CAF) phase with 
an extra nonzero order {vijZ ■ [Si x Sj)) ^ between 
next nearest neighbor sites. Unlike the Neel-|-nematic 
order, the CAF order has a different symmetry from the 
pure Neel order even with presence of the QSH spin-orbit 
coupling in the background. For instance, when the Neel 
vector is along the y direction, it is still invariant under 
the PH transformation defined in Eq. El but the CAF or- 
der breaks this PH transition, as was discussed in section 
IIIB. 

This symmetry analysis leads to the following two con- 
clusions: 

(1). If there is a CAF phase with zero A, there must be 
a transition line within the inplane Neel phase in Fig. [5f) 
even with finite A (dashed line of Fig. [2f)), which corre- 
sponds to the order-disorder transition of the CAF order 
parameter. 

(2). The transition between the QSH insulator and the 
Neel-|-CAF phase is not a 3d XY transition, because the 
symmetry breaking at this transition is different from the 
3d XY transition. 

The different predictions between our theory and 
Ref.^^ can be checked numerically in the future. 
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